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ABSTRACT 

The purpose of this study was to determine if administrative 
aspirants from preparation programs, experienced school administrators, and 
groups of individuals with no connection to education would perform 
differently on a short vignette discontinued by the Educational Testing 
Service (ETS) but still used as a sample of the assessment in the School 
Leaders Licensure Assessment (SLLA) . The SLLA is used by an increasing number 
of states as a determinant of initial eligibility for a state certificate in 
educational administration. Six subject groups participated: (1) educational 

leadership graduate students; (2) Leadership Academy participants (graduate 
students); (3) school superintendents; (4) Army National Guard members; (5) 
business students; and (6) graduate psychology students. In all there were 53 
educators and 5.6 noneducators. Findings do not indicate that the SLLA stamps 
a person as administrative material, in that a number of successful school 
superintendents performed poorly. The test did discriminate well between 
those who understood the Interstate School Leaders' Licensure Consortium 
{ ISLLC) standards and those- who did not, and this is the stated purpose of 
the assessment. To do well on the SLLA, the respondent needed to know 
something about the ISLLC standards. Whether the SLLA serves the states that 
use it is a question requiring further study; findings do suggest that a 
state that adopts the SLLA as a screen for administrators may be narrowing 
its pool of administrators. (Contains 1 table and 16 references.) (SLD) 
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Alternative Paths to Administrative Practice: The New School Leader’s Licensure 

Assessment 

Introduction and Background 

This study compares and contrasts the responses of members of six groups to a 
discontinued item used previously in the School Leaders Licensure Assessment (SLLA) 
developed by the Educational Testing Service (ETS). The SLLA is used by an increasing 
number of states as a means of determining initial eligibility for a state certificate in 
educational administration. 1 The SLLA is an entry level assessment designed to capture 
an administrative aspirant’s knowledge of the Interstate School Leader’s Licensure 
Consortium’s (ISLLC) standards. The six cores areas that the ISLLC consortium 
members believed were essential factors for a beginning school administrator were 
identified as follows: 

• Having a vision 

• Having a focus on student learning 

• Providing for a safe learning environment 

• Encouraging parent and community involvement 

• Acting with integrity and fairness 

• Understanding the large political, social, economic, legal and cultural contexts 

of schooling. 

ETS developed the SLLA as a means of assessing whether or not an 
administrative aspirant possesses a reasonable knowledge of these six essential standards. 
Each of the written prompts or queries or vignettes on the SLLA was written to 
incorporate one or more of these six ISLLC standards. The creation of the standards was 
done intentionally to drive the practice of educational administration in new directions 
(Hessel & Holloway, 2002). It appears that this consortium has had some success in 
impacting the preparation of school administrators. It has now morphed into a new group 
called the Educational Leadership Constituent Council (ELCC). If one doubts the 
potential influence of the ELCC on administrative preparation, consider a recent posting 
on the ELCC website: 

1 Since last year’s AERA meeting, the state of California has entered into a contract with ETS. 




3 



SLLA Paper 3 



The Educational Leadership Constituent Council (ELCC) is 
constituency group comprising the American Association of 
School Administrators (AASA), the Association for Supervision 
and Curriculum Development (ASCD), the National Association 
of Elementary School Principals (NAESP), and the National 
Association of Secondary School Principals (NASSP). The 
purpose of this council is to review university-based educational 
administration programs that seek NCATE (National Council for 
the Accreditation of Teacher Education) accreditation using 
national standards developed by the National Policy Board for 
Educational Administration (NPBEA). The ELCC confers 
"national recognition ” to those programs that meet these 
standards. (ELCC, 2003) 

To some degree, ELCC has become the enforcer of these standards, the one’s we seek 
to better understand by examining the SLLA. To some degree, the professional 
associations are exerting an unusual pressure on professors of educational 
administration. That this pressure has political muscle is due in part to the beliefs that 
the SLLA does capture essential skills and attitudes necessary for the successful new 
administrator. For a cogent critique of this development in administrator preparation, 
see English (2003) in an article entitled, Functional foremanship and the virtue of 
historic amnesia: The AASA, the ELCC Standards, and the reincarnation of Scientific 
Management in educational preparation programs for profit. 

Purpose 

The purpose of the study was to determine if administrative aspirants from 
preparation programs, experienced school administrators, and groups of individuals with 
no connection to education would perform differently on a short vignette discontinued by 
ETS but still used as an example of the assessment. The authors of this study reasoned 
that individuals who had been trained in an educational administration preparation 
program should perform at higher levels on the SLLA assessment than those who come 
to the assessment with little background in education. The purpose of the research was to 
test that reasoning. With respect to demonstrated proficiency on a SLLA vignette, is 
there a significant variation in group scores depending upon group membership? Do the 
scores of those study participants who have a background and/or preparation in education 
and or administration differ significantly from those with no administrative background 
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or preparation? And, do those versed in education earn more Is and 2s than those with 
no connection to educational practice? 

Theoretical Base 

The SLLA is a unique response to the challenges that states and school districts 
face as they seek to expand and improve the pool of potential new school administrators 
(Bryant, Hessel & Isemhagen, 2002). The SLLA is designed to identify”those candidates 
who possess the knowledge and skills believed to be important for competent, beginning 
level professional practice” (Schmitt, 1995). As such the assessment joins a large family 
of instruments and tests designed to capture one's aptitude and ability to do a job. From a 
broad perspective, the SLLA and the philosophical foundation upon which it is based— 
the ISLLC standards-represent an attempt to dramatically modify the educational 
administration profession. Hessel and Holloway (2002) presented the ISLLC standards 
as “in the service of rebuilding" or “reculturing the leadership infrastructure of 
schooling.” Murphy (2002), in the annual yearbook of the National Society for the Study 
of Education, echoed Hessel and Holloway with a call for “reculturing the profession.” 
These were lofty and far reaching goals. 

The field of educational administration has, over the years, been the target of 
numerous calls for reform (Cubberly, 1923; Newlon, 1934; Callahan, 1962; McLaughlin, 
1984; Gibboney, 1987; Griffeths, 1987). Many of these calls for reforms were focused 
on inputs, on the curriculum content, relevant teaching experience, and corollary learning 
that went into the making of the school administrator. Some reform efforts have 
supported changing the organization and structure of school administration by nurturing 
new and less hierarchical leadership skills proposed as necessary alteration in how school 
administrators behave. A good example would be the site based management movement 
of the late 70s and early 80s (Pierce, 1980). Another example is the effective schools 
movement with its focus on leadership at the building level. Others have sought to infuse 
a critical perspective into the preparation and practice of school administration (Foster, 
1986: Bryant, 1988-99; Peca, 2000). 

The efforts of ISLLC should be understood within this historical frame of 
reference. The practice of school administration has not been static and has been 
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repeatedly the target of efforts to change it. The ISLLC reform effort can be understood 
within the context of the broad accountability movement in education. 2 In recent years, 
school reformers have sought to make the parts of the educational system more 
accountable for outcomes. The assessment and testing of students has increased 
dramatically. The use of teacher assessment has also increased. In most states, school 
districts are measured on the basis of aggregate test scores. With the SLLA, the focus on 
accountability now shines on school administration. 

This new examination of outcomes was given a prominent stage in a recent 
yearbook publication of the National Study of Education. Repeated arguments were 
made in favor of examining the dispositions and skills of those moving into educational 
administration. Much of the burden for insuring accountability in the ranks of those hired 
new to the profession of educational administration has been shouldered by ETS and the 
SLLA. It is this assessment that is gaining prominence across the country. Given these 
high expectations for dramatically changing the profession of educational administration, 
research on what this assessment actually does measure is of critical importance. Given 
the adoption of this assessment by a growing number of state departments of education as 
a screen for administrative employment, the SLLA has quickly become a high stakes 
assessment 

How well do those trained in educational administration programs perform on the 
assessment? Does the assessment and its standards based core really serve as a measure 
of ability and capacity to lead a school? Does the test discriminate well between those 
who understand the ISLLC standards and those who do not? Does one really need to 
know anything about the specific ISSLC standards in order to do well on the SLLA 
assessment? Does the SLLA really serve well those states that now require a certain pass 
or cut score as a necessary pre-requisite for the administrative license? Many questions 
surface when high expectations are loaded upon a high stakes test. But there is little 
empirical examination of the SLLA and of those who complete the assessment. 

Livingston described in one of the earliest descriptions of the SLLA a pilot of the 
assessment. In reporting on the process of piloting the assessment, Livingston noted that 
white and minority participants differed in their group scores (Livingston, 1998). Van 



2 Most recently ISLLC has been discontinued and reconstituted as ELCC. 
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Meter and McMinn presented a description of the assessment along with instructions 
about how to use it (2001). Holloway (2002), one of the architects of the SLLA, 
presented an argument that the SLLA effectively bridges the gap between theory and 
practice. Anderson (2002) argued equally persuasively that the SLLA is shallow and 
“promotes a narrow view of instructional leadership” (p. 70). Bryant, Hessel and 
Isemhagen (2002) also described the SLLA and reported on its implications for the field. 

The SLLA represents a different kind of reform effort. Unlike most educational 
administration reform efforts, the SLLA focuses on outcomes and behaviors. It is 
designed to serve as a tool for determining who possess adequate skills, knowledge, and 
dispositions to serve in the schools and who does not. And, as many states have now 
adopted a satisfactory score on the SLLA as a certification requirement, the assessment 
becomes an instrument for reforming the manner in which educators assume positions in 
educational administration. Indeed, if, as has been the case in some states, one may be 
licensed to be a school administrator with no administrative training providing one earns 
a satisfactory score on the SLLA, a longstanding tradition of preparing new 
administrators in formal educational administration programs may disappear. 

The implications of the SLLA and its adoption by states are huge. Yet, limited 
research has examined the performance of practicing administrators on the SLLA. A 
theme journal of The Journal of Personnel Evaluation in Education was devoted to the 
ISLLC standards and several articles to the SLLA (Ellett, 1999). In this edition, Reese 
and Tannenbaum provided evidence that the SLLA did adequately test for knowledge of 
the ISLLC standards (Reese and Tannenbaum, 1999). McCown et al.(2001) found that 
practicing principals were rated by their superintendents as quite knowledgeable of the 
ISLLC standards. Coutts (1997) looked to see if principals who failed were singularly 
uninformed about the ISLLC standards. The superintendents in his study felt these 
unsuccessful principals were not well versed in the ISLLC standards. Anderson (2002) 
presented an argument that the component pieces of the SLLA represented a focus that 
was too narrow and not sufficiently comprehensive. The study reported in this paper 
represents one of the few efforts utilizing empirical evidence in looking critically at the 
SLLA. 
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Research Design 

The team of researchers identified six different groups of subjects willing to 
participate in the study by writing a response to a discontinued item from the SLLA. 
These subjects were recruited in Nebraska, Kansas, and California. One of the parts of 
the SLLA consists of a series of short vignettes or mini case studies with an imbedded 
problem that is tied to one or more of the ISLLC standards. Those taking the SLLA 
assessment construct written responses to these vignettes. The written responses are then 
scored by readers trained to evaluate the responses according to rubrics developed by 
ETS. We replicated this process used by ETS in conducting this study. While our 
numbers are small and our process diminutive in contrast with that followed by ETS, the 
steps we followed were as follows: 

1) Identification of subject groups; 

2) Administration of the discontinued ETS item 

3) Identification of Scorers 

4) Training of Scorers 

5) Scoring of Items 

6) Construction and Analysis of Data 
Identification of a Subject Group 

Drawing upon local resources each member of the research team located groups 
willing to participate in the study. There were three groups located in Nebraska, two in 
Kansas, and one in California. We wanted groups of subjects who were 1) enrolled in or 
recently graduated from a program in educational administration, 2) administrators with 
substantial experience, and 3) individuals with little or no connection with education. We 
also wanted a group of individuals who were clearly planning to become educational 
administrators as these would form a group that might logically be expected to sit for the 
SLLA assessment at some point in the future. 

Administration of the Item 

Permission was granted by Educational Testing Service to use an item that had 
been discontinued. We used the following item: 

In March a high school senior presents a letter from his mother requesting, contrary to school 
policy, that he be allowed to drop physics, because he is failing the class. He is also failing 
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several other classes, but he does not need to pass physics to graduate. The principal consults 
with the teachers and with the student's counselor. They all concur that the student could be 
passing all his courses, including physics, if he worked harder. However, the principal, 
persuaded by the parent s argument that the stress of physics is adversely affecting her son 
authorizes the student to drop the course . 

Evaluate the principal's a ction from the point of view of the learning and teaching . 

We had concerns not about the item but about the prompt. ETS had discontinued the 
item for a reason and it appeared to us that the prompt asked for a neutral evaluation 
whereas the normal SLLA queries or prompts asked for the respondent to indicate what 
she or he would do. But, ETS has proprietary rights to their assessment material and this 
was the item we were granted permission to use. Furthermore, it is an item that appears 
on some websites as a sample of the SLLA assessment. We felt it necessary to use the 
item exactly as it was given to us. 

This vignette was identified by ETS as linking to the ISLLC standards of 1) 
professional development that promotes a focus on student learning; 2) treating 
community stakeholders equitably; and 3) acting with integrity and fairness. Presumably, 
this vignette raises issues that have to do with focusing on the individual student’s 
learning, making sure to recognize and honor the parent’s position, and acting in a fair 
manner to the student and the teachers. 

Subjects were assembled in a classroom setting. A standard protocol was 
developed that was used in each administration of the item. The researcher would read 
the instructions, hand out a mutual consent form and the item and request the response to 
be completed within five minutes. Several subjects complained that this was not long 
enough but most were able to communicate their response to the prompt. 

Identification of Scorers 

Following the pattern ETS uses to locate individuals to score the assessment, we 
identified four individuals to score the assessment. There were all individuals with 
administrative experience: 

Scorer One: a former school principal of many years, the former executive 
director of a national educational administration association, and currently a professor of 
educational administration. 
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Scorer Two: a former school superintendent and retired professor of education 
and associate dean of a college of education, and former executive director of an 
international educational association; 

Scorer Three: a current school superintendent in a medium sized school district; 

Scorer Four: a current assistant principal in a secondary school and doctoral 
student in an educational administration program. 

With these individuals we were able to replicate the practice of ETS in using as 
scorers those individuals well versed in educational practice. 

Training of Scorers 

We met with these scorers prior to asking them to read the responses in order to 
familiarize them with the rubrics developed by ETS that would need to be applied to the 
assessment. We were careful to emphasize that, in scoring these responses, we were only 
looking for evidence of the ISLLC expectation called for in the rubric. Matters of 
grammar, writing coherence, educational beliefs, and expressions of ideas unrelated to 
the rubric were to be ignored. This is in keeping with the training ETS provides to its 
scorers. It is one aspect of the SLLA that led Anderson (2002) and English (2003) to 
question its validity as a high stakes test. 

Scoring of Items 

In keeping with assessment practices developed by ETS, scoring rubrics were 
developed based on these ISLLC standards as follows. Two scorers rated each response 
as a 2, 1 , or 0 based on the following: 

2 = This response is concerned primarily with what is in the best interests of this 
particular student. In addition the response cites any two of the following: 

• conferencing with the parent who may have essential information about 
the student; 

• conferencing with the student to help the student confront and begin 
to solve the problem; 

• involving other appropriate staff members to address possible 
causes/reasons for failure; 

• generating a plan of action that will provide support to the student; 

• working toward parent/student cooperation with the school, and their 
acceptance of responsibility for achieving passing grades in all other 
courses. 
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1 - The response is supportive of what is in the best interest of this particular 
student, citing any one of the following: 

• conferencing with the parent who may have essential information about 
the student; 

• conferencing with the student to help the student confront and begin 
to solve the problem; 

• involving other appropriate staff members to address possible 
causes/reasons for failure; 

• generating a plan of action that will provide support to the student; 

• working toward parent/student cooperation with the school, and their 
acceptance of responsibility for achieving passing grades in all other 
courses. 

0 = Response is vague, or omits reference to any of the essential factors. 

If there was a disagreement in the scores of the two readers, two members of the 
research team read the response and discussed it and made a determination about a final 
score. There was a high level of correspondence between scorers. The inter-rater 
reliability of the four scorers using Kendall’s tau was calculated as (.845, Sig .000, 
n= 139). 

Construction and Analysis of Data 

Data from the 141 responses included both demographic data and the outcome 
results of the scoring. We were able to use 139 valid cases in most instances. We elected 
to use SPPS in analyzing the data and to use the Chi Square test of independence in 
answering our research questions. For example, we hypothesized that those in or recently 
in administrative preparation programs would have higher scores (more Is and 2s) than 
those in non-educator groups. We hypothesized that experienced administrators would 
also have a greater proportion of Is and 2s than non-educators. 

Description of Groups 

Because our method required an across group analysis (the group was our unit of 
analysis), we describe the characteristics of each group below and present descriptive 
statistics of the groups in Table One. We present the groups in the order that we 
anticipated the relationships of group score to group membership. That is, we 
hypothesized that the first three groups would have more Is and 2s than the last three 
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groups and that our listing of the groups would parallel a rank ordering of groups by 
SLLA score. 

Ed Leadership Graduate Students (1 ) 

The research subjects in our first group belonged to an emerging 
administrators program that meets on one Saturday a month. We labeled this group 
Ed Leadership Graduate Students. All were participating in a voluntary initiative to 
help them learn about administration. Many were enrolled in a masters program in 
educational administration from different institutions in the state. Most planned to 
seek positions in educational administration. A few already had administrative 
positions. The assessment was administered on January 18, 2003. 

Leadership Academy Participants (2) 

The research subjects in our second group were participants in a Leadership 
Academy. All were graduate students in an educational administration preparation 
class. All were actively studying educational administration and most were planning 
to earn a masters degree. 

School Superintendents (3) 

The research subjects in our third group were all superintendents. One of these 
was a central officer administrator and the other 28 served mainly rural school districts 
within the Midwest. All but one were male. Most were in their 40s or 50s with a 
handful outside this range. Seventy-six percent held a master’s degree and 24% held a 
doctorate. Most (62% had worked as a teacher for 10 years or less although one had 
taught for 29 years. Approximately a third had been in an administrative position for 
5 to 10 years. None were newly certified as administrators. They responded to the 
vignette on February 27, 2003 at a statewide workshop. 

Army National Guard (4) 

The research subjects in this were members of the Nebraska National Guard. 
Several had experience in education. All had managerial responsibility in the Guard. 
Because access to members of the National Guard is strictly controlled, these 
individuals were selected by one member of the research team based on convenience. 
Based on the findings we report below, we believe it appropriate to identify a larger 
sample of such individuals. The assessment was done on January 20th and 21 st . 
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Business Students (5) 

The research subjects in this group were participants in a university 
undergraduate business class. There were both traditional and non-traditional 
students in the class. These subjects were taught by an educational administrator who 
was willing to ask the students to participate in the SLLA study. There were 34 
participants. The assessment was administered on 12/12/03. The research team 
member who administered the item wrote of this group: “The group is diverse— all 
over 21 up to the age of 50+. Some have HS diplomas and are working on BAs. 

Some have a great deal of work experience; others have none.” It was a group with 
little connection with p-12 education. 

Graduate Psychology Students (6) 

These members of this group were willing subjects enrolled in a graduate 
psychology class at a Midwestern university. As was true of the previous group, most 
had little connection to education. But, they were more homogeneous as a group. All 
had completed a bachelor’s degree and were currently seeking a master’s degree in 
some field within psychology. Slightly more than 40% listed themselves as full time 
students and 35% held jobs in organizations that provide mental health or social 
welfare services. Seventy percent of this group were in their 20s. The item was 
administered on Dec. 2, 2002. 

Research Questions 

In order to determine if there were significant differences across groups, we 
analyzed the distribution of scores (0s, 1, 2s) using the Chi Square test of independence. . 
Our hypotheses were that subjects in Emerging Administrators and the Leadership 
Academy and the Superintendents groupings would have a greater number of Is and 2s 
than those in the other three groupings. 

Our main research question was as follows: 

1) Is there a significant difference in scores on the SLLA based on group 
membership? We used the Chi Square test of independence to answer this question. 

We were also interested in several other questions as a part of this study. In 
addressing these questions we formed new groups for analysis. The first was our 
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anticipation that educators would score more Is and 2s on the SLLA than would non- 
educators. 

2) Is there a significant difference in scores on the SLLA based on whether or not 
one works in an educational field? Again, we used the Chi Square test of independence 
to determine differences across just two groups: 1) all educators in our study (n = 53) and 
2) all non-educators (n = 56). Our hypothesis was that educators should have higher 
scors on the SLLA item than non-educators. 

Finally, as an item of interest, we wanted to know whether there was a 
difference in those who anticipated a shift to administration and those who did not. 

That is, we felt that since many of these subjects (n = 59) indicated no interest in ever 
being an educational administrator, perhaps these individuals would receive more Os 
and fewer Is and 2s than those who indicated they were interested in becoming 
administrators (n = 30). 

Results 

Descriptive statistics for the study population appear in Table One. One 
critical dependent variable is the SLLA Score variable presented last in the table. 



Table One About Here 

The data in Table One indicate some interesting visual differences relative to the 
scores achieved by the six different groups on the SLLA. For example we anticipated 
more Is and 2s on the item for the experienced group of school administrators. We did 
not anticipate such a high number of Is and 2s for the Army National Guard officers. 

Table Two compares the performance of the different groups on the SLLA. 
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Table Two 



Percentage Results of Scores bv Groupings 



Os Is 2s 



Emerging Administrators (N = 25) 
Leadership Academy ( N = 26) 
Superintendents (N = 29) 

Army National Guard (N = 10) 
Business Students (N = 34) 
Psychology Students (N = 17) 



40% 48% 12% 

42% 27% 31% 

79% 14% 07% 

50% 20% 30% 

61% 27% 12% 

76% 12% 12% 



To some degree, the percentages displayed in Table Two follow our predictions. We felt 
that study participants in groups that were actively studying educational administration 
would have a stronger knowledge of student centered concerns since instructional 
leadership is a strong focus of educational administration programs. Hence, the 
Emerging Administrators and the Leadership Academy subjects both reveal a greater 
percentage of Is and 2s than any other group. We were surprised at the percentage of 0s 
recorded by the Superintendents and by the greater than predicted percentages of the 
Army National Guard Participants. For the Business Students and the Psychology 
students, the scores were as we anticipated. Most of these subjects, unfamiliar with the 
values imbedded in the SLLA item, did not construct responses that could be judged as 
connected to the rubrics. 

In looking at the scores across groupings, the Chi Square test of independence 
achieved significance. The results appear in Table 3. 

Table 3 

Chi Square Test of SLLA Scores bv Groupings 

Value df Asymp. Sig 
(2-sided) 

Chi Squre 22.046* 10 .015 

N Valid Cases 139 



* 8 cells have expected count of less than 5; minimum expected is 1.51. 
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We conclude from this significant statistic that not all groups are equal when it comes to 
the SLLA. There is a relationship between group membership and the scores on the 
SLLA and these scores do not vary independently. 

We were also interested in testing whether or not subjects in various educational 
positions would achieve higher scores than subjects outside fields of education. 
Accordingly, we created two groups: Educators and Non-Educators. Table Three 
presents descriptive data on the scores for these two groups. 

Table 3 

Crosstabulations of Scores by Educators vs. non-Educators 



Score 


All Educators All non-Educators 


Row Totals 


0 Count 
% Total 


45 

(32.8%) 


36 

(26.3%) 


81 

(59.1%) 


1 Count 
%Total 


23 

(16.8%) 


12 

(8.8%) 


35 

(25.5%) 


2 Count 
(15.1%) 


13 

(9.5% 


8 

(5.8% 


21 

(15.3%) 


Column 


81 

(59.1%) 


56 

(40.9%) 


137 

(100%) 



We noted that in Table 3 a simple count indicates a higher percentage of Is and 2s in the 
Educator group (26.3%) than in the non-Educator Group (14.6%). Table 4 presents our 
Chi Square analysis. 



Table 4 

Chi Square Test of Educators vs non-Educators 

Value df Asymp. Sig 
(2-sided) 

ChiSqure 1.123* 2 .570 

N Valid Cases 137 

*0 cells have expected count of less than 5; minimum expected is 8.58. 

The results of the Chi Square test of independence indicated no relationship between 
score and group membership. We wondered if the unusual distribution of scores among 
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the 29 superintendents might mask differences so we removed this group from the 
analysis. Table 5 contains cross tabulations for the scores of this new group. 



Table 5 

Crosstabulations of Scores by Educators vs. non-Educators with 



Superintendents 



Removed 



Score 


All Educators All non-Educators 


Row Totals 


0 Count 


23 


36 


59 


% Total 


(21.1%) 


(33.0 %) 


(54.1%) 


1 Count 


19 


12 


31 


%Total 


(17.4%) 


(11.0%) 


(28.4%) 


2 Count 


11 


8 


19 


(15.1%) 


(10.1% 


(7.3%) 


(17.4 %) 


Column 


53 


56 


109 




(48.6%) 


(51.4%) 


(100%) 


The data in 


Table 5 parallel 


our descriptive results above with the Educator group 


earning a greater percentage of Is and 2s (27.5%) than the non-Educator group (18. 
But the Chi Square test for independence failed to achieve significance as shown in 



6 . 



Table 6 

Ch i Square Test of Educ ators vs nonEducators with Superintendents Removed 

Value df Asymp. Sig 
(2-sided) 

ChiSqure 4.840* 2 .089 

N Valid Cases 109 

*0 cells have expected count of less than 5; minimum expected is 9.24. 

Our last statistical analysis focused on whether or not those who anticipated 
becoming a school administrator earned higher scores on the SLLA than those who 
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indicated no interest in doing so. Table 7 presents descriptive data on the scores of these 
two groupings. 

Table 7 

Crosstabulations o f Scores by Administrative Aspirants vs non- Aspirants 



Score 


Aspirants 


Non- Aspirants Row Totals 


0 Count 
% Total 


11 

(12.3.%) 


39 

(43.8%) 


50 

(56.2 %) 


1 Count 
%Total 


12 

(13.4%) 


13 

(14.6 %) 


25 

(28.1%) 


2 Count 
(15.1%) 


7 

(7.8% 


7 

(7.8%) 


14 

(15.7%) 


Column 


30 

(33.7%) 


59 

(66.2%) 


89 

(100%) 



Clearly, a greater percentage of those who have no interest in administrative careers score 
at lower levels on the SLLA than those who plan on administrative practice. This 

descriptive statistic is supported by a significant Chi Square in Table 8. 

Table 8 

Chi Squar e Test of Educators vs nonEducators with Superintendents Removed 

Value df Asymp. Sig 
(2-sided) 

Chi Squre 22.944* 2 .000 

N Valid Cases 89 

*0 cells have expected count of less than 5; minimum expected is 29.7. 



Summary 

Our statistical analyses indicate that across our initial groupings the following 
possible conclusions about the SLLA: 

1) If one is in a program such as the Emerging Administrator Program or a 
Leadership Academy type program in which training is tied to the prevalent 
beliefs about practice, one is likely to score better on the SLLA. If one is a 



SLLA Paper 1 8 



practicing administrator of many years of experience, that experience appears 
to be unrelated to one’s performance on the SLLA. 

2) Differences in performance on the SLLA disappear when examining 
differences between educators and non-educators. 

3) Differences between those who plan on administrative careers and those with 
no interest in such careers are plain. Those administrative aspirants are better 
able to provide answers that satisfy the rubrics of the SLLA than those who do 
not aspire to administrative positions. 

Because there is clearly some aspect of group membership at work in these statistical 
results we scanned the actual responses to the items for qualitative themes that might help 
us better understand these results. We report these themes in the next section. 

Qualitative Themes 

While it was not our original intent to analyze the responses qualitatively, we did 
scan the responses for what ISLLC refers to as “dispositions”. By this term we were 
looking to see if we could identify any particular set of values that appeared common to 
the responses in each of the six groupings. Dispositions is a word not clearly defined in 
the ISLLC literature but can certainly be understood to include what the NASSP 
Assessment Center referred to as “educational values.”(Bryant, 1 990). Each of the 
researchers read through all of the responses and identified themes independently. We 
found agreement among us for the following kinds of dispositions. 

In the Emerging Administrator Group and the Leadership Academy group there 
was a tendency to focus on the interests of the student. Such responses would receive 
positive evaluations by the scorers. Examples of this focus can be found in the following 
representative statements: 

The pressure to maintain a good GPA, or class rank for scholarship reasons may 
be a factor. ” 

“/ think the principal is try (sic) to keep the parent happy and not looking out for 
the best interest of the child. If there was concern about this child, it should have been 

addressed by the school administrator with the parent before the parent brought it to the 
school’s attention. ” 
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“I feel the principal should have met with the parent, student, teacher , and 
counselor to discuss ways to assist the student with passing classes. 

“By dropping physics the child will be able to concentrate on all his other 
classes. ” 

Comments of these kinds were judged by the scorers to be "in the interests of the 
student", one of the dominant values in the ISLLC standards. Another theme in these two 
groups was that of developing some kind of a plan to address the needs of the student: 

/ am wondering if a different solution could have been arrived at in order to 
satisfy all parties; one that may involve a unique grading system, a creative arrangement 
of resources for the student. 

Conferencing with parents and students and teachers was interpreted as an 
inclusive action that would lead to planning, another ISLLC value. 

It is interesting, however, to look at the different ways that one can interpret what 
"in the interest of the student" might mean. For some of the psychology students, the 
principal was enabling the student to duck a responsibility. 

The principal is helping to develop the student ’s tendency to stop trying and 
enabling the student to give up on working on grades. 

A number of the respondents in this group may have ultimately seen the principal as 
working for the welfare of the student but the manner in which they expressed this 
concern was indirect and did not result in high scores on the SLLA. A typical response 
was: 

“teaching the student that if something is too tough he can just quit or get 
someone else to get him out of work that is too difficult for him. 

The scorers found that such a response simply did not connect with the rubric about a 
focus on the best interests of the student. This is in part because the training required that 
they not ascribe meaning to a response but look only at the words on the paper. 

Interestingly, a number of the subjects in the Business student group shared this 
idea of that allowing a student to take an "easy way" out of a problem might not be in the 
best interests of the student. One wrote: 

"This would discourage the student from working harder. This will create a habit 
for the student not to try hard in able to be successful. " 
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Another subject in the Business student group suggested: 

"The student in wanting to drop the class was looking for the easy way out, which 
is what the educational system should not endorse. " 

The superintendents formed one of the most interesting groups. Nine of the 23 
who scored a 0 viewed the vignette from the perspective of policy. Ignoring a school 
policy set a dangerous precedent in the eyes of these subjects. They also echoed a theme 
common to the psychology students group. They were critical of the implicit message in 
the principal’s decision: when the going gets tough, it is okay to quit. 

Discussion 

We organize our discussion about the larger questions we posed at the beginning 
of this paper. 

Does the SLLA assessment serve as a measure of ability and capacity to lead a school? 

If we accept the proposition that the SLLA stamps a person as administrative 
material, our answer is no. We have a number of very successful school superintendents 
in our subject pool that performed poorly on the SLLA. However, as the framers of the 
SLLA note often, it is an assessment designed not to measure the ability of a test taker to 
administer a school district. The SLLA is designed to measure knowledge of the ISLLC 
standards. Why, we ask, are state policy makers then using this assessment as a screen 
for certification? Why, we ask, are some department of educational administration using 
the SLLA as a test of administrative aptitude? 

Does the test discriminate well between those who understand the ISLLC standards 
and those who do not? 

Our answer to this question is yes. Based on our data, the SLLA does 
discriminate. Those involved in administrative preparation activities that are current and 
have a focus on instructional leadership and student learning appear to earn higher scores 
than those with no knowledge of current educational ideology and practice. And, those 
whose preparation is ongoing or recent scored better than veteran superintendents who 
had little or no knowledge of the ISLLC standards. However, we would note that there is 
significant issue relative to the lens one uses to define what the “best interest” of the 
student is. For the Emerging Administrators and Leadership Academy, the concept of the 
“interests of the student” was defined according to the rubrics. For the Business 
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Students, the Superintendents, and the Psychology students, “best interests” often meant a 
different interpretation, one centered on remaining firm behind an action that has the 
physics student “toughing it out.” 

Does one really need to know anything about the ISLLC standards to do well? 

Our data suggests an affirmative answer to this question. Those who were aligned 
with the practices of instructional leadership, student achievement, and student centered 
decision making, integral values in the ISLLC standards earned higher scores. 

Does the SLLA serve well the states that require it? 

We do not believe we can answer this question from with our present data. To 
serve well, the SLLA will have to result in a “reculturing of the profession” that does lead 
to better school administrators. Will schools have better principals if the preparation of 
school administrators focused on these standards? The jury is out on this question. 

English (2003) for example believes that these standards de-contextualize administrative 
decision making, rendering what is complex and situational into the algorithmic and 
simple. 

Conclusions 

When a state adopts the SLLA as a screen for administrators, it runs the risk of 
narrowing its pool of administrators to only those who understand the ISLLC standards 
(or ELCC standards). Is this wise? There are many attributes to the successful school 
administrator. No one has yet determined that a high score on the SLLA is correlated 
with a measure of successful practice. That is the missing piece and is one that needs to 
be explored. 
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Table One 

Descriptive Statistics by Groupings (Actual Numbers) 





Emerging 

Administrators 


Leadership 

Academy 


Superintendents 


National 

Guard 


Business 

Students 


Psycholog; 

Students 




N = 25 


N = 26 


N = 29 


N= 10 


z 

II 

U> 

4 ^ 


N= 17 


Male 


10 


11 


28 


9 


20 


5 


Female 


15 


15 


1 


1 


14 


12 


Last Degree 














High Sch 


0 


0 


0 


0 


32 


0 


BA/BS 


5 


3 


0 


4 


2 


17 


MA/MS 


19 


23 


22 


3 


0 


0 


PhD/EdD 


1 


0 


7 


1 


0 


0 


Other 


0 


0 


0 


2 


0 


0 


Profession 














Teacher 


15 


7 


0 


1 


0 


0 


Admin 


2 


13 


28 


0 


0 


0 


Other Ed 


8 


4 


1 


0 


2 


3 


NonEd 


ry 


2 


0 


8 


32 


14 


Plans 














No 


0 


1 


0 


8 


34 


17 


Yes 


20 


11 


0 


1 


0 


0 


Already 


5 


14 


29 


0 


0 


0 


Score 














Os 


10 


11 


23 


5 


20 


13 


Is 


12 


7 


4 


2 


9 


2 


2s 


2 


8 


2 


3 


4 


2 
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